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Program Evaluation 

Program Evaluation at the Department of State: FY 2012 Accomplishments 

The Department of State (DOS) and US AID have made major progress since release of the Quadrennial 
Diplomacy and Development Review report in December 2010 to institutionalize a process of monitoring 
and evaluation at both agencies. US AID put in place a new evaluation policy in January 201 1 and the 
Department did likewise in February 2012. 

Since the implementation of its new evaluation policy, the Department has aggressively moved forward 
on efforts to build a foundation for the use of evaluation findings to inform: a) the establishment or 
revision of the Agency's strategic objectives; b) budgetary and programmatic decisions; and c) strategies 
that support the use of evaluations and performance data (e.g., indicators) to improve Agency decision- 
making. 

In FY 2012, the Department focused implementation of the evaluation policy at the bureau level, i.e., at 
bureaus based in Washington, DC (roll-out of an evaluation policy for posts is planned for 2013). 
Progress was made in FY 2012 on three major fronts: capacity building; supporting rigorous, high-quality 
evaluations of programs, projects, initiatives, approaches, etc.; and development of two-year Bureau 
Evaluation Plans (BEP) tied to a bureau's strategic objectives. The following is a synopsis of 
accomplishments in FY 2012: 

Capacity Building. The Department developed and provided interim evaluation training to regional, 
functional and management bureau staff to strengthen their understanding of evaluations and capacity to 
plan and budget for evaluations. The interim training served as a precursor to professionally-developed 
training courses that became available in FY 2013 under the auspices of the Department's Foreign 
Service Institute (FSI). In addition, a 100-person DOS Evaluation Community of Practice (CoP) 
representing more than 30 DOS bureaus and USAID staff meets monthly to share standards and best 
practices and serves as a forum for working through complex evaluation issues. 

Supporting High-Quality Evaluations. The Department awarded five Indefinite Delivery/Indefinite 
Quantity (IDIQ) contracts to facilitate contractual services for the evaluation of the Agency's diplomatic 
and development efforts. While emphasizing the importance of independently-conducted evaluations, an 
objective of implementation of the IDIQ (as well of capacity building efforts) is to help bureaus determine 
the most rigorous study designs appropriate for their bureaus' programs/projects/efforts given their size, 
stage of development and other factors. In addition, the Department issued comprehensive evaluation 
guidelines on the planning, managing, and conduct of evaluations. Both the evaluation policy and 
evaluation guidelines stress the rigor and independence of performance and impact evaluations — the two 
principal types of evaluations carried out by the Department's bureaus. 

Bureau Evaluation Plans (BEPs). The Department's evaluation policy requires all bureaus to put in place 
a Bureau Evaluation Plan that describes two to four evaluations to be completed by FY 2014. Bureaus 
submitted BEPs in the Spring 2012 to the Directors of Budget and Planning (BP) and the Office of U.S. 
Foreign Assistance Resources (F) proposing 100 evaluations to be completed. These 100 evaluations 
represent a 500% increase over FY 201 1 and include evaluations for economic statecraft, PEPFAR, 
security initiatives, domestic passport workload management, conflict stabilization operations, and rule of 
law programs, among others. BEPs are informed by the bureau's strategic objectives as outlined in the 
Joint Regional Strategy (for regional bureaus) and the Functional Bureau Strategy (for functional and 
management bureaus). 
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Next Steps for the Department 

The evaluations underway, combined with ongoing implementation of the evaluation policy, have begun 
to instill a culture of evaluation envisioned by the QDDR. Further implementation efforts in FY 2013 
include the roll -out of comprehensive training on an ongoing basis in the form of two FSI-supported 
courses: "Managing Evaluations" and "Evaluation Designs and Data Collection Methods." Second, the 
DOS Evaluation Community of Practice will help guide implementation of a study in FY 2013 on the 
evaluation of "diplomacy" (defined as the pursuit of U.S. national interests through political, peace 
building, economic, environmental and cultural spheres). Third, the CoP will work with the Directors of F 
and BP on development of evaluation policy for posts. Preliminary work on evaluation policy for posts 
was initiated in FY 2012. 



These and other implementation strategies are positioning the Department to more effectively plan and 
budget for, implement, and make active use of evaluations for Agency decision-making. 

Management Challenges 

In its FY 2012 annual assessment, the Department's Office of Inspector General (OIG) identified the most 
serious management and performance challenges for the Department to be in the following areas: 

1 . Protection of People and Facilities 

2. Contract and Procurement Management 

3. Information Security and Information Management 

4. Financial Management 

5. Military to Civilian -Led Transitions — Iraq and Afghanistan 

6. Foreign Assistance Coordination and Oversight 

7. Diplomacy with Fewer Resources 

8. Public Diplomacy 

9. Effective Embassy Leadership 

10. Consular Operations 

The OIG's assessment, which can be found on pages 146-155 of the FY 2012 Agency Financial Report, 
was based on its review of recent information from a variety of sources including reports done by OIG, 
GAO and Congressional Committees. In response to recommendations contained in the reports, the 
Department's bureaus and offices took a number of corrective actions. Information on actions taken and 
actions remaining on the challenges can be found on pages 156-168 of the FY 2012 Agency Financial 
Report. The most recent version of the Agency Financial Report can be found at the following website: 
http://www. state. gov/s/d/rm/c6113. htm 

Discontinued Indicators 



As discussed in the upfront section entitled. Selection Criteria for Performance Indicators", the 
Department of State has shifted to more outcome -oriented performance measurement and adopted 
SMART performance criteria for developing and selecting performance measures for the Annual 
Performance Plan. For FY 2012, due to changes in the planning and budgeting process the following 
listed 38 indicators with performance data are proposed for discontinuation in the FY 2012 Annual 
Performance Report. To view actual performance for every discontinued indicator, see: 
http://www.state.gov/documents/organization/203415.pdf There are 38 indicators proposed for 
discontinuation: eleven from Strategic Goal 1, eleven from Strategic Goal 3, one from Strategic Goal 4, 
three from Strategic Goal 5, one from Strategic Goal 6, and eleven from Strategic Goal 7. 
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